In the method of analyzing a sequence of two-dimensional images containing a plurality of independently moving rigid bodies to ascertain three-dimensional information about the moving bodies, various prior art techniques are available for readily identifying features or points associated with the rigid bodies common to the image frames. Similarly, once all the features or points of a selected rigid body have been identified in successive image frames, prior art techniques are available for providing the desired three-dimensional information about the selected rigid body.
However, there remains a need for a better process for determining which features or points in the successive image frames are associated with which rigid body in the images. The present invention is directed to this need.
Prior art processing assumed to have identified features or points common to all image frames. The present invention permits determination of which features or points are associated with which rigid body in an image containing a plurality of such rigid bodies.
A camera records an image comprising a plurality of independently moving rigid bodies. The motions are relative to the camera and hence, the camera determines a fixed coordinate system. If the scene includes a "background" and the camera is moving, the background will appear as another object moving relative to the camera.
In accordance with the teachings of the present invention, two-dimensional projections are generated by a set of rigid bodies moving freely in three-dimensional space. The projections are specified by the mathematical product of matrices with very special structural properties determined by the quantity of bodies and the degree of independence of their respective motions. The present invention concerns the regenerating of those structures from the set of two-dimensional observations.
More particularly, in the field of image processing it is well known to apply feature detection algorithms and techniques to each image frame in order to identify and locate points or features common to the image in each frame. For example, each corner of an object in an image frame can be identified and its coordinate in the image frame determined.
Moreover, it is known in the art how to determine position and velocity of objects from a sequence of image frames, such as are obtained from a camera, video recorder or the like.
The present invention provides a vital link between the "front end" feature detector and the "back end" processor of object position and velocity. That is, a computer used to practice the present invention receives as its input, data of the feature points and their coordinates in each image frame, usually in the form of electrical signals from standard feature detectors, such as the one described by Tomasi and Kanade in a report entitled "Detection and Tracking of Point Veatures", Tech Report CMU-CS-91-132, Carnegie Mellon University, School of Computer Science, Pittsburgh, Pa., and provides at its output data indicative of which of the detected feature points are associated with a respective moving body in a sequence of images containing a plurality of such moving bodies for further application dependent processing of the trajectories of each moving body.
The invention has application in such fields as collision avoidance. For example, a camera located in the front of a vehicle, which may be either stationary or moving, records images of the scene. In the case of an automobile traveling on a roadway, the images might include a background, road signs, other vehicles travelling on the road and so forth. The sequence of images frames are pre-processed by a standard feature detector. The resultant data is provided, in accordance with the present invention, to a general purpose computer where the detected points are grouped as to specific features or bodies in each image frame. The data as to which points are associated with which moving body in the image are provided, for example, to a conventional collision accordance system in order to determine which, if any, moving body will collide with the vehicle, which moving bodies are moving away from the vehicle or which moving bodies are moving in unison with the moving vehicle. Also determinable are which objects form a part of the background or off-the-road objects which are not a threat to the vehicle. In the absence of feature grouping, the mere detection of feature points will not result in proper collision accordance.
In a similar manner, the invention has applicability to robot vision systems, for example, for distinguishing among multiple moving objects traveling past the robot. The invention is also useful in robot vision systems for collision avoidance of a moving robot. Another application is for grouping or distinguishing image points of a robot hand as contrasted with the image points of an object being manipulated by the hand.
The above examples are merely illustrative of the applications to which the invention may be applied. It will be apparent to those skilled in the art that the present invention has utility and application in many other image processing applications.
In each of these illustrative examples, the present invention provides a means for receiving data from a conventional feature detector indicative of detected feature points and processing the data in order to group detected points as arising from a respective moving body in an image containing a plurality of such moving bodies, so that additional signal processing may be performed on the processed data based upon the points associated with each moving body in the image frames.